Computing vocal entrainment: A signal-derived PCA-based quantification scheme with application to affect analysis in married couple interactions

In human–human interactions, entrainment is a naturally occurring phenomenon that happens when interlocutors mutually adapt their behaviors through the course of an interaction. This mutual behavioral dependency has been at the center of psychological studies of human communication for decades. Quantitative descriptors of the degree of entrainment can provide psychologists an objective method to advance studies of human communication including in mental health domains. However, the subtle nature of the entrainment phenomenon makes it challenging for computing such an effect based on just human annotations. In this paper, we propose an unsupervised signal-derived approach within a principal component analysis framework for quantifying one aspect of entrainment in communication, namely, vocal entrainment. The proposed approach to quantify the degree of vocal entrainment involves measuring the similarity of specific vocal characteristics between the interlocutors in a dialog. These quantitative descriptors were analyzed using two psychology-inspired hypothesis tests to not only establish that these signal-derived measures carry meaningful information in interpersonal communication but also offer statistical evidence into aspects of behavioral dependency and associated affective states in marital conflictual interactions. Finally, affect recognition experiments were performed with the proposed vocal entrainment descriptors as features using a large database of real distressed married couples’ interactions. An accuracy of 62.56% in differentiating between positive and negative affect was obtained using these entrainment measures with Factorial Hidden Markov Models lending further support that entrainment is an active component underlying affective processes in interactions. © 2012 Elsevier Ltd. All rights reserved.